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The  optimist  fell  from  a  ten  story  window; 
As  he  passed  each  bar, 

He  shouted  to  his  friends  inside, 

"I'm  alright  so  far." 
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PREFACE 


This  paper  was  prepared  as  a  contribution  to  Government  Policies 
for  Industrial  Innovation:  Design,  Implementation,  Evaluation  (New 
York:  Associated  Faculty  Press,  1984),  edited  by  Prof.  J.  David 
Roessner  and  sponsored  by  the  Policy  Studies  Organization.  It 
represents  a  major  revision  of  my  earlier  Rand  Paper  (P-6919),  The 
Evaluation  of  Innovation ,  in  terms  of  clearing  up  earlier  ambiguities  of 
concept  and  definition  and  including  significant  amounts  of  new 
material . 


EVALUATING  PROGRAM  INNOVATION: 
A  POLICY  PERSPECTIVE 


Introduction 

Like  our  proverbial  optimist,  the  innovator- -be  it  an  individual  or 
an  organization- -is  ever  sanguine  that  this  particular  innovation  is  (or 
shortly  will  be)  successful.  The  conditions  which  motivate  the 
innovation- -some  sort  of  recognized  and  important  shortcoming  or 
opportunity- -have  spurred  and  formed  the  innovation  activity,  producing 
a  program  which  is  directly  tailored  to  meet  the  immediate  exigency. 
There  should  be  little  doubt,  then,  that  the  innovation's  objectives 
will  be  met,  given  the  appropriate  time  and  resources.  Interim 
shortfalls  will  prove  transitory  and  be  corrected  as  the  full  effect  of 
the  program  takes  hold.  Or  so  the  system  proponent  would  have  one 
believe. 

But  we  know  empirically  that  this  is  little  more  than  a  naive 

belief.  Uncertainty  pervades --indeed,  characterizes — the  technology 

innovation  processes.  Technological  and  procedural  innovations  can  fail 

for  any  number  of  reasons,  even  when  the  causes  are  predictable  and 

understood.  Sometimes  the  extant  technology  cannot  support  the  promise, 

either  technically  or  economically,  regardless  of  the  resources 

invested.1  Often  times,  when  the  technology  is  in  hand,  the 
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institutional  conditions  are  hostile.  And,  finally,  even  the  occasional 

felicitous  junction  of  technology,  process,  and  organizational 

receptivity  leaves  innovation  and  the  expected  diffusion  wanting  because 
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of  a  change  in  the  ambient  conditions. 

From  an  analytic  perspective,  the  success  or  failure  of  innovation 
is  maddeningly  paradoxical.  In  examining  both  technical  and  procedural 
innovations,  researchers  can  consistently  and  accurately  detail  where 
and  why  failures  have  occurred.  The  reasons  for  specific  failures  are 
known.  From  these,  policy  recommendations  have  been  dram  and 
innovative  means  of  correcting  shortcomings  and  generally  promoting 
innovation  are  prescribed  (Varner,  1974).  Yet  these  too  are  just  as 
failure  prone.  Closer  examination  reveals  the  reason:  the  particular 
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circumstances  that  define  the  innovative  context  are  so  unique  as  to 
defy  ready  comparison  with  other  innovation  arenas,  at  least  in  a 
prescriptive  mode.  Evaluations  of  technical  innovations  are  erroneously 
superimposed  upon  managerial  innovations;  little  distinction  is  made 
between  evaluating  the  technology  and  the  technology  delivery  system. 
Wider  samples  based  upon  aggregate  comparisons  are  usually  statistically 
inconclusive  (Ettlie,  1982).  The  "lessons"  drawn  from  one  set  of 
innovations  may  be  only  problematically  applicable  to  another,  if  at 
all;  intra-industry  comparisons  are  problematic  and  inter-industry 
transfers  are  dubious.  Indeed,  the  solutions  to  the  first 
organization's  problems  might  actually  be  counterproductive  in  regard 
to  the  second.^ 

If  the  empirical  record  on  innovation  were  not  sufficiently 
discouraging,  its  intellectual  underpinnings  are  equally  (if  not  more 
so)  deficient,  perhaps  even  foreboding.  Mowery  (1983)  has  recently 
argued  the  inherent  inadequacies  of  the  economic  theory  of  innovation, 
with  special  emphasis  on  Arrow's  appropriability  model.  Political 
science  has  difficulty  limning  the  interaction  between  the  public  and 
private  sectors  regarding  innovation:  where  does  one  begin  and  the 
other  end,  especially  as  the  two  increasingly  become  inseparable 
partners?^  What  knowledge  we  do  have,  claim  Nelson  and  Winter  (1977: 
41),  "is  in  the  form  of  congeries  of  semi-isolated  facts,  rather  than  a 
connected  intellectual  structure."  In  sum,  while  the  total  weight  of 
research  is  surely  hefty,  "such  theory  as  exists  is  poorly  developed  and 
insufficiently  tested."  (Mogee,  1980:  189) 

But  if  innovation  itself  is  difficult  and  quixotic,  the  demands  for 
its  benefits  are  even  more  difficult  to  allay  or  temper,  especially  in 
today's  highly  competitive  commercial  environments.  The  problem  is 
hardly  limited  to  specific  firms  or  industrial  sectors;  the  National 
Academy  of  Science  warns  that  technological  development  must  be 

...among  the  highest  national  priorities  of  the  nation.  The 
United  States  must  act  now  ...  (because]  the  nation's  capacity 
for  technological  innovation  is  vulnerable  both  from  domestic 
weaknesses  and  from  damaging  practices  of  other  nations.  To 
lose  out  in  this  competition  could  be  extremely  damaging 
- (Boffey,  1983) 
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Moreover,  in  a  period  of  public  and  private  recession  and  declining 
budgets,  analysts  are  being  urged  to  innovative  means  to  salvage  their 
organization’s  services  (Walker  and  Chaiken,  1982).  The  insistent 
question,  therefore,  is  not  whether  we  can  predict  the  ultimate  success 
or  failure  of  an  innovation,  nor  (more  fundamentally)  if  one  should  even 
attempt  to  innovate  given  the  opportunity  costs  incurred  as  well  as  the 
direct  attributable  costs  of  failure.  As  an  article  of  intellectual 
faith  and  policy,  innovation  activities  will  and  should  be  encouraged 
and  facilitated.  The  more  immediate  question  addressed  here  is  how  does 
one  evaluate  on-going  innovation  activities,  so  that  their  progress  can 
be  most  effectively  managed,  their  end  result  most  expeditiously 
obtained,  and  the  final  product  most  beneficially  received  by  both 
sponsors  and  recipients. 

This  essay  addresses  the  latter  set  of  questions,  leaving  the  first 
to  those  with  a  longer-range ,  more  teleological  cast.  The  particular 
concern  here  will  be  the  evaluation  of  innovation  while  it  is  on-going, 
for  it  is  during  this  period  that  critical  choices  must  be  made.  A 
second  emphasis  will  be  on  the  policy  implications  of  such  evaluation 
activities,  especially  as  they  apply  in  the  public  sector.  The 
objectives,  then,  are  twofold:  first,  to  encourage  a  more  integrated 
understanding  of  innovation  processes  and,  second,  to  propose  specific 
ways  in  which  these  processes  can  be  evaluated  to  improve  our  management 
of  innovation  midstream.  These  goals  are  based  on  two  underlying 
assumptions:  first,  that  it  is  preferable  to  evaluate  innovation  while 
it  is  evolving  rather  than  permit  it  to  continue  unhindered  to  a 
possible  final  failure;  and,  second,  that  even  in  light  of  the 
discouraging,  even  dismal  record,  there  are  some  optimistic 
possibilities,  especially  in  the  program  evaluation  literature,  towards 
these  objectives  (Levine  et  al.,  1981;  Rossi  and  Freeman,  1982). 

The  focus  of  this  essay  is  on  the  evaluation  of  individual 
programs,  as  it  must  be  to  describe  and  recommend  evaluation 
requirements.  Evaluating  grand  national  strategies  of  innovation  (e.g., 
the  "^industrialization"  of  the  United  States)  is  deliberately  beyond 
its  intended  scope,  although,  taken  in  their  aggregate  form,  the 
evaluation  of  individual  programs  could  have  implications  for  assessing 
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the  larger  policies.  More  realistically,  however,  such  strategies 
entail  cotoplexities  surely  beyond  our  current  conceptual  capabilities  to 
prescribe  and  implement,  let  alone  assess  (Brewer  and  deLeon,  1983: 

Chap.  9)  with  any  confidence  in  mid-course  corrections. 

McLaughlin  tellingly  describes  the  evaluation  of  innovation  as 
"alchemy,"  arguing  that  our  present  "logic  of  inquiry  is  wrong.  And 
preoccupation  with  scientism  and  with  fixing  our  traditional  evaluation 
paradigms  scants  what  we  do  know....  One  major  challenge  for 
evaluators,  then,  is  epistemological:  to  develop  new  and  valid  ways  of 
knowing."  (McLaughlin,  1980:  46)  The  alchemy  analogy  is  fitting,  for 

at  this  stage  of  both  the  innovation  and  evaluation  arts  and  crafts,  the 
dogmatic  adherence  to  scientific  rigor  or  holistic  approaches  would  be 
premature. ^  Rather,  this  essay  deliberates  what  we  already  know  in  hopes 
of  better  utilizing  existing  knowledge  and  evidence.  Thus,  it  adopts  a 
deliberately  heuristic  spirit  to  provide  directions  and  milestones 
towards  the  evaluation  of  innovation  rather  than  a  complete  and 
dependable  mapping. 
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Definitions,  Examples,  and  Recommendations 

One  of  the  inherent  problems  of  studying  innovation  is  that  it  has 

assumed  a  numbing  variety  of  divergent  definitions.  Furthermore, 

innovation  can  occur  in  a  number  of  different  arenas. ^  Most  conanonly, 

however,  it  can  be  categorized  as  either  technical  or  procedural, 
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although  the  distinction  is  hardly  hard  and  fast.  The  first  generally 
refers  to  the  research,  development,  and  demonstration  (RD&D)  of  a  new 
technology,  with  the  final  success  or  failure  of  the  technology  assessed 
as  a  function  of  its  commercial  acceptance.  Often  this  is  dichotomized 
into  product  and  process  innovations.  The  procedural,  or  what  some  have 
termed  behavioral,  innovation  deals  less  with  technology  and  more  with 
management  and  institutions.  For  example,  Abernathy  and  Rosenbloom 
(1982)  argue  that  both  American  and  Japanese  firms  were  presented  with 
identical  technological  opportunities  in  the  development  and 
commercialization  of  consumer  electronics  but  that  the  Japanese 
managerial  (i.e.,  procedural)  strategies  provided  them  with  the 
innovative  edge  over  their  U.S.  rivals. 

Evaluation  refers  to  the  assessment  of  a  system,  generally  in  a 
retrospective  mode,  in  terms  of  its  performance  towards  meeting 
specified  objectives.  For  present  purposes,  evaluation  is  primarily 
defined  as  a  means  for  assessing  and  managing  the  innovation  process. 
Most  of  the  work  on  evaluating  innovation  has  been  in  the  first  category 
(i.e.,  technology),  as  analysts  apply  the  standard  criteria  of  technical 
feasibility,  development  time,  and  projected  cost.  There  is,  however,  a 
growing  awareness  that  while  such  measures  are  necessary,  they  are  not 
sufficient  for  predicting  innovation  results  (deLeon,  1982).  Technical 
and  economic  measures  must  be  matched  with  procedural  metrics.  In 
short,  an  expanded  set  of  evaluative  criteria  must  be  proposed,  tested, 
and  applied  if  one  is  to  assess  with  any  confidence  the  innovative 
process  while  it  is  still  on-going.  Lacking  this,  the  ability  to 
correct  the  continuing  process  becomes  more  questionable  than  we  can 
afford. 

A  first  order  consideration  is  our  fundamental  perception  of  the 
innovation  process  and  how  it  shapes  our  expectations  and  analyses.  As 
alluded  to  above,  an  organization  or  individual  probably  does  not  even 
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begin  R&D  without  expectant  visions  of  a  successful  outcome;  psychic  and 
financial  costs  rarely  permit  such  serendipitous  excursions.  For 
instance,  financial  return  is  typically  cited  as  the  primary  cause 
underlying  a  firm's  decision  to  risk  investing  in  unproven  technologies 
(Mansfield,  1968).  Most  innovation  models  and  prescriptions  are 
therefore  based  on  and  biased  towards  a  premonition  or  presumption  of 
success.  Yet  we  know  statistically  that  the  great  majority  of  attempted 
innovations  ultimately  end  in  failure  or,  at  least,  in  achievements  well 
below  initial  expectations.  These  failures  usually  are  recognized  only 
after  heavy  capital,  institutional,  and  personal  investments;  early, 
easy  terminations  are  the  exception  rather  than  the  rule  (Myers  and 
Sweezy,  1978).  The  rare  success  (e.g.,  semi-conductors)  seductively 
warps  our  view  of  the  innovation  process  and  its  much  less  prevalent 
selection  of  new  technologies.  A  SPRU  (1972)  study,  which  explicitly 
compared  innovation  successes  with  failures,  couched  its  recommendations 
in  terms  of  promoting  the  former  as  opposed  to  avoiding  the  latter. 

While  this  is  perhaps  understandable,  the  emphasis  creates  one  critical 
disadvantage.  To  study  mainly  successes  severely  limits  our  knowledge 
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of  why  things  fail.  As  failure  is  certainly  the  most  frequent  outcome, 
we  are  implicitly  precluding  the  modal  condition  from  our  model.  One 
needs  to  understand  and  compare  both  success  and  failure  conditions  to 
assess  policies  as  they  relate  to  the  technology  innovation  process. 

One  of  the  major  conceptual  obstacles  hindering  the  evaluation  of 
R&D  and  many  innovations  is  that  their  respective  processes  are  often 
indiscriminately  lumped  and  treated  as  an  on-going,  seamless  progression 
in  which  scientific  theory  leads  to  applied  technology  leads  to 
innovation  and  then  diffusion.  Rates  of  innovation  may  differ  but  the 
process  is  viewed  as  an  integrated  entity.  Even  admitting  to  an 
occasional  glitch  or  hitch  in  the  system,  this  depiction  underlies  much 
of  the  technology  assessment  philosophy  and  diffusion  of  innovation 
literature.  It  is,  1  propose,  a  detrimental  myth,  at  best  an  infrequent 
phenomenon  one  should  be  hesitant  to  adopt  as  a  policy-relevant  model. 
Rather,  I  would  strongly  advocate  the  analytic  disaggregation  of  the 
innovative  process.  It  is  more  accurate  and  useful  to  identify  discrete 
stages  and  judge  then  as  a  function  of  their  own,  more  focused 
desiderata.  However  one  may  choose  to  demarcate  the  innovative  process, 
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it  should  be  clear  that  different  actors  with  different  objectives  and 
criteria  play  different  roles  during  the  innovation/diffusion  drama.  To 
force  them  simultaneously  onto  the  same  tableau  might  be  theoretically 
plausible,  but  it  ultimately  disguises  their  individual  contributions 
and--more  to  the  point--our  ability  to  evaluate  their  particular 
effects.  For  instance,  the  development  and  application  of  new 
technologies  to  urban  environments  has  been  a  primary  concern  of  the 
federal  government,  local  municipalities,  and  many  private  sector 
firms. Each  has  the  same  general  objective — the  delivery  of  better, 
more  efficient  services  to  a  local  clientele--but  this  blanket  objective 
obscures  more  concrete,  differentiated  goals  and  criteria.  In 
delivering  improved  health  or  personal  security  services,  the  federal 
government  may  be  sponsoring  new  technologies  under  the  broad  ic*«date  of 
public  goods;  the  municipality  might  be  reacting  more  to  local  rising 
costs  (or  decreasing  revenues)  and  union  demands;  and  the  private  sector 
participants  could  well  be  responding  strictly  to  a  profit  motive. 

While  none  of  these  stands  in  forced  opposition  to  each  other,  they  are 
sufficiently  divergent  that  to  render  an  easy  and  accurate  evaluation  of 
an  innovative  technology  and  its  accompanying  delivery  system  is  very 
difficult,  especially  when  both  are  still  evolving. 

If,  at  the  formative  stages  of  innovation,  evaluation  criteria  and 
milestones  could  be  established  for  the  various  participants  and 
periods,  one  would  assuredly  have  a  much  better  possibility  of  assessing 
the  probability  of  a  successful  (or  deficient)  innovation.  For  example, 
early  in  the  innovative  activity,  technical  difficulties  may  prove  the 
main  obstacles  and  technical  correctives  accorded  primary  importance. 
Much  later  in  the  activity,  the  delivery  of  the  RD&D  product  may  be  the 
critical  impediment  and  procedural  remedies  may  be  more  appropriate. 
Although  both  problems  are  common  to  the  innovative  process,  each 
requires  different  measures  and  responses  to  be  overcome. 

The  stage  distinction  might  be  important,  but  it  is  not  easy  to 
identify  or  resolve,  either  on  a  conceptual  or  operational  basis.  Even 
if  one  were  to  pose  sharply  distinguishable  stages  in  theory,  in 
practice  they  ebb  and  flow  indistinguishably,  one  into  the  other. 
Institutional  inertia,  the  uncertain,  risky  nature  of  R&S,  and  the 
problematic  transitions  from  actor  to  actor  reinforce  the  incremental 
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nature  of  the  successful  innovation. **  As  McLaughlin  (1980:  44) 

advises:  "the  process  of  implementation  is  heuristic--a  process  of 
learning  and  adjusting,  rather  than  a  process  of  installation." 
Furthermore,  an  undifferentiated,  unswerving  adherence  to  such 
milestones  could  undercut  the  innovative  dynamic,  for  early  deficiencies 
could  conceivably  be  overcome  in  later  stages  if  the  RD&D  were  allowed 
to  proceed;  conversely,  early  optimism  scarcely  guarantees  later 
fruition.  There  is,  in  a  sense,  an  underlying  innovation  Darwinism 
implied  here.  Yet  it  should  be  clear  that  one  cannot  realistically 
assess  innovation  in  process  unless  critical  milestones  are  marked  and 
observed  and,  furthermore,  such  winnowings  are  necessary  as  the  costs  of 
innovating  (or  not  innovating)  become  increasingly  great.  It  is  not  an 
easy  balance  to  strike  and  maintain. 

This  first  proposal  is  relatively  specific  in  its  practical 

applications.  It  calls  for  evaluation  programs  custom-tailored  to  their 

given  innovation  contexts:  the  technical  and  procedural  palliatives 

which  measure  and  apply  to  the  development  of  an  SST  do  not,  ipso  facto, 

apply  to  medical  technology,  and  even  less  to  behavioral  or  managerial 

innovations.  My  second  suggestion  is  more  concrete  but  could  have  a 

much  more  significant  effect  on  our  ability  for  innovation  evaluation  as 

a  generic  activity.  Simply,  the  presently  available  data  are  inadequate 

for  the  current  and  escalating  needs.  Even  if  we  admit  to  the 

shortcomings  in  the  existing  body  of  theory- -which  of  course  undermine 

the  data  definition  and  collection  efforts--one  is  still  struck  by  the 

paucity  of  reliable  data  that  are  comparable  over  a  large  number  of 

cases.  Branscomb  (1983:  134)  has  observed  that  "the  general  perception 

is  that  you  cannot .. .expect  to  find  existing  reliable,  evaluated  data, 

even  on  matters  you  know  have  been  subjected  to  scientific  research  and 

publication."  If  we  lack  the  ability  to  compare  and  contrast  among 

incidents  of  innovation,  then  we  are  hardpressed  to  move  up  the 

evaluation  learning  curve.  Too  often  one  finds  methodological 

sophistication  being  substituted  for  empirical  validity;  the  end  result, 

while  perhaps  epistemologically  elegant,  almost  certainly  does  not 

satisfy  the  immediate  public  policy  objectives  and  probably  does  not 

advance  the  state  of  the  evaluation  art  because  of  the  unreliable 
12 

empirical  inputs. 
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While  the  observation  "if  you  think  our  theory  is  bad,  wait  'til 
you  see  our  data"  night  be  obvious  to  most,  the  mere  recognition  of  the 
problem  does  not  portend  its  quick  relief  for  at  least  three  reasons. 
First,  as  noted  above,  lacking  some  consensual  theory  of  innovation, 
data  requirements  are  difficult  to  formulate  a  priori  (Downs  and  Mohr, 
1976);  one  is  confronted  with  the  inevitable  chicken  and  egg  conundrum. 
Second,  individual  innovation  cases  can  be  so  distinctive  as  to  require 
very  different  measurement  data.  NASA-sponsored  technology  programs 
might  legitimately  be  judged  quite  differently  than  those  sponsored  by 
the  Department  of  Commerce.  And  finally,  data  do  not  grow  naturally,  as 
on  analytic  trees,  to  be  harvested  conveniently  when  succulent;  an 
organic  part  of  the  program,  they  are  rarely  independently  defined  and 
cultivated  for  the  sole  analytic  convenience  of  the  evaluator.  Too 
often,  researchers  "erroneously  focus  their  attention  on  access  and 
distribution  rather  than  on  the  scholarship  required  to  put  data  in  a 
form  in  which  users  will  dare  rely  on  them."  (Branscomb,  1983:  134) 

For  all  these  reasons,  the  data  problems  loom  imposingly  against  the 
evaluation  of  innovation  while  it  is  an  on-going  endeavor. 

But  this  pessimistic  litany  is  not  necessarily  a  counsel  of 
despair.  Rather,  it  is  more  an  admonishment  of  restraint,  of  more 
realistic  and  attainable  objectives.  We  should  not  be  blinded  or 
obsessed  by  the  immediate  relevancy  of  case  studies  nor  the  long-range 
goal  of  general  theory  (Kaplan,  1964;  Hirschman,  1976).  At  the  current 
state  of  knowledge,  usable  typologies  of  innovation  would  be  greatly 
beneficial  and,  I  submit,  directly  applicable  in  the  evaluation  of 
innovations  from  a  public  policy  perspective.  Just  as  important,  the 
development  of  such  typologies  would  not  only  advance  the  state  of  the 
methodological  art  by  highlighting  what  we  know  and  do  not  know  (i.e., 
some  cells  will  be  empty),  but  they  are  obtainable.  For  instance,  one 
might  wish  to  relate  the  interactions  of  "demand-pull"  and  "capability- 
push"  R&D  strategies  with  different  "selection  environments"  (Nelson  and 
Winter,  1977).  If  the  development  of  such  typologies  could  produce  such 
results,  then  it  would  go  a  great  way  in  alleviating  the  data  problem 
presented  above  as  well  as  making  innovation  evaluation  more  germane  and 
accessible  to  decisionmakers. 
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This  observation  leads  in  a  cumulative  manner  to  a  final 

consideration:  the  question  of  audiences  and  their  objectives.  This 

essay  has  an  operating  assumption  that  the  evaluation  of  innovation  is 

inherently  more  than  an  intellectual  exercise,  however  stimulating  that 

exercise  might  be.  The  implicit  motivation  is  to  improve  the  innovation 

process,  that  is,  to  provide  both  public  and  private  organizations  with 

a  substantive  understanding  of  "policy  concerns  [which]  arise  with 

respect  to  both  the  effects  on  the  development  in  question  and  the 
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effects  of  that  development."  For  this  reason,  the  information  and 
analysis  must  be  made  salient  and  accessible  to  innovators  and  the 
pertinent  decisionmakers  in  a  form  they  can  use. 

This  observation  is  a  logical  deduction  from  the  earlier  two  points 
in  terms  of  three  policy  implications.  First,  evaluation  must  directly 
address  the  policy  problem  if  it  is  to  pass  the  essential  relevancy 
test.  This  lends  support  to  the  tailored,  specific  approach  as  well  as 
the  movement  towards  a  more  generic  typology.  It  is  true  that  "The 
experimental  paradigm,  a  useful  tool  in  medicine  and  social  psychology, 
where  treatment  is  discreet  and  standard,  cannot  be  fixed  to  fit  the 
reality  of  planned  change  in  social  systems  such  as  public  education," 

(McLaughlin,  1980:  45)  but  it  is  possible  that  lessons  gleaned  from 
public  education  studies  could  be  applicable  to  research  on  job  training 
programs  or  other  social  programs.  Second,  to  present  persuasive  policy 
recommendations,  better  data  definition,  collection,  and  analysis  must 
be  incorporated.  This  implies  that  "data  grubbing"  must  be  initiated 
from  the  very  outset  of  the  program  so  that  they  will  fit  the  particular 
analytic  requirements  throughout  the  entire  program  for  the  complete  set 
of  participants.  Lacking  these,  the  analyst  will  have  a  difficult  time 
convincing  the  policymaker  of  the  accuracy  and  relevancy  of  the 
assessment,  let  alone  what  corrective  measures  should  be  adopted. 

Third,  and  perhaps  most  challenging,  is  the  education  of  program 
managers  as  to  the  benefits  of  program  evaluation  and  a  greater 
appreciation  of  the  more  sophisticated  tools  that  are  available  and 
relevant  to  their  needs  (Clarke,  1974).  While  this  is  hardly  a 
straightforward  assignment ,  it  is  essential  if  the  evaluator's  skills 
are  to  influence  the  innovation  processes. 
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Some  Final  Observations 

I  have  argued  here  that  the  evaluation  of  the  in-process  innovative 
dynamic  is  a  difficult  but  increasingly  necessary  policy  endeavor.  This 
characterization  applies  in  both  its  conceptual  and  practical  aspects. 

If  we  are  to  gain  intellectual  and  policy  leverage  on  it,  then  it  is 
imperative  to  move  in  discernible,  attainable  steps  to  limited,  mid¬ 
range  goals.  The  fact  that  they  are  limited  in  their  immediate  purposes 
does  not  mean  that  they  are  limited  in  their  utility  (deLeon,  1982). 

Specifically,  I  proposed  as  first  steps  a  disaggregation  of  the 

innovation  process  as  a  unit  of  analysis  and  the  development  of  an 

innovation  typology.  These  are,  of  course,  not  novel  suggestions.  What 

distinguishes  them  from  existing  models  is  their  immediate  and  explicit 

connection  to  policy  considerations,  either  in  the  private  or,  more 

likely,  the  public  sector.  These  recommendations  momentarily  neglect  a 

number  of  critical  components  delineating  how  one  gets  from  here  to 

there,  such  as  the  bases  for  structuring  the  typology,  the  criteria  for 

differentiating  within  the  innovative  process,  or  the  appropriate  level 

of  policy  attention.  But  the  purpose  of  this  essay  is  more  to  lay  out 

broad  boulevards  of  policy-relevant  evaluation  rather  than  provide  the 

building  brick  and  mortar.  Direction  and  purpose  are  as  important  as 

the  final  destination  at  this  stage  of  our  knowledge.  A  research  corpus 

already  exists  which  could  be  used  as  a  starting  point;  Bransccnb  (1983) 

has  proposed  a  "national  science  add  technology  data  policy";  specific 

examples  of  RD&D  stages  can  be  cited  which  are  at  least  illustrative  and 
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might  serve  as  postulated  models  for  stages  in  innovation.  In  terms  of 
extant  methodologies  and  computational  powers,  we  are  already  equipped 
well  beyond  our  intellectual  capabilities. 

The  need,  I  submit,  is  not  for  more--or  even  better--case  studies 
of  innovation  nor  for  the  development  of  more  rigorous  analytic  tools, 
for  neither  directly  addresses  the  necessary  comparative,  cross -industry 
innovation  activities,  nor  how  one  evaluates  the  process.  Ve  are 
already  case  study  glutted  and  methodologically  muscle-bound.  The  more 
persistent  needs  are  to  restructure  our  intellectual  and  empirical 
understanding  of  innovation,  largely  using  the  existing  evidence  and 
tools.  For  instance,  we  should  conceptualise  innovation  as  a  complex, 
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nonlinear  phenomenon*^  rather  than  treat  it  as  a  natural, 
monodirectional ,  rather  systemic  condition  located  somewhere  on  a 
learning  curve.  This  approach  suggests  new  ways  of  perceiving  phenomena 
without  demanding  new  data  or  methodological  breakthroughs. 

This  essay  has  implied  that  there  are  positive,  potential  benefits 
to  be  derived  from  accurate,  timely  evaluations  of  innovation.  In 
closing,  we  should  also  note  the  negative  side  of  the  same  coin,  the 
opportunity  costs  wasted  by  poor  evaluation  in  terms  of  failing  to 
terminate  unwarranted  projects  until  they  become  obvious.  These  "costs" 
can  be  economic  (cost  overruns),  technological  (the  neglect  of  other, 
more  promising  technologies),  or  political  (a  public  skepticism 
regarding  technology*^) .  As  Porter  and  Rossini  (1983:  728)  remind  us, 

"Society  cannot  afford  to  ignore  the  possible  costs  attendant  on 
technological  development,  even  as  it  continues  to  seek  its  bounties." 
For  both  the  positive  and  negative  reasons,  we  are  thus  moved  to  the 
difficult  task  of  having  to  review  (and  possibly  revise)  the  innovation 
drama  while  it  is  still  in  production,  before  the  final  denouement  has 
occurred.  This  charter — although  not  easily  fulfilled  from  an  analytic 
perspective — and  the  conditions  alluded  to  above  should  motivate  the 
evaluator  of  the  on-going  Innovation  process  with  sufficient  cause  (some 
might  claim  urgency)  to  explore  in  greater  detail  some  of  the  notions 
presented  above. 
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ENDNOTES 


The  liquid  metal  fast  breeder  reactor  has  yet  to  overcome  its 
technical  problems  and  fulfill  the  glowing  commercial  promise  ascribed 
to  it  over  30  years  ago.  A  recent  analysis  of  LMFBR  problems  is 
Lanouette  (1983). 


2 

The  difficulty  experienced  in  establishing  budgetary  innovations  such 
as  PPBS  and  Zero  Based  Budgeting  is  explained  by  Wildavsky  (1979,  Chap. 
6). 

3 

Even  with  demonstrated  technologies  and  generous  government  subsidies, 
the  widespread  diffusion  of  solar  energy  systems  has  been  severely 
arrested,  at  least  partially  because  of  the  public  perception  that  the 
energy  crises  which  characterized  the  1970s  were  only  passing  phenomena. 

A 

Identical  education  innovations  have  had  very  different  effects  when 
introduced  in  different  school  districts;  see  the  multiple  volumes 
authored  by  Berman  et  al.  (1975). 

5 

A  comprehensive,  if  now  somewhat  dated,  review  of  this  relationship  is 
Pavitt  and  Walker  (1976). 


And  unnecessary;  see  Kaplan  (1964). 

^  Much  of  this  is  comprehensively  reviewed  by  Mogee  (1980). 

p 

Both  must  be  concomitantly  considered;  for  instance,  see  Kidder 
(1981). 
o 

deLeon  (1978)  offers  some  possible  explanations  for  this  emphasis. 

See  Lambright  et  al.  (1979)  and  Roessner  (1979). 

**  These  are  stressed  by  Nelson  and  Winter  (1977). 

12 

A  ready  charge;  for  instance,  see  Acland  (1979). 

**  The  distinction  is  made  by  Porter  and  Rossini  (1983:  728);  emphasis 

in  original.  Cf.  Lasswell  (1971),  who  distinguishes  between  knowledge  in 
and  of  the  policy  process. 

**  For  example,  Baer  et  al.  (1977). 

**  This  description  was  a  major  (although  certainly  not  surprising) 
finding  stressed  by  Nelson  and  Langlois  (1983);  also  Mogee  (1980). 
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